Global and Quadratic Convergence of Newton Hard-Thresholding Pursuit

Algorithms based on the hard thresholding principle have been well studied with sounding theoretical guarantees in the compressed sensing and more general sparsity-constrained optimization. It is widely observed in existing empirical studies that when a restricted Newton step was used (as the debiasing step), the hard-thresholding algorithms tend to meet halting conditions in a significantly low number of iterations and are very efficient. Hence, the thus obtained Newton hard-thresholding algorithms call for stronger theoretical guarantees than for their simple hard-thresholding counterparts. This paper provides a theoretical justification for the use of the restricted Newton step. We build our theory and algorithm, Newton Hard-Thresholding Pursuit (NHTP), for the sparsity-constrained optimization. Our main result shows that NHTP is quadratically convergent under the standard assumption of restricted strong convexity and smoothness. We also establish its global convergence to a stationary point under a weaker assumption. In the special case of the compressive sensing, NHTP effectively reduces to some of the existing hard-thresholding algorithms with a Newton step. Consequently, our fast convergence result justifies why those algorithms perform better than without the Newton step. The efficiency of NHTP was demonstrated on both synthetic and real data in compressed sensing and sparse logistic regression

A semismooth newton method for the nearest Euclidean distance matrix problem

The Nearest Euclidean distance matrix problem (NEDM) is a fundamentalcomputational problem in applications such asmultidimensional scaling and molecularconformation from nuclear magnetic resonance data in computational chemistry.Especially in the latter application, the problem is often large scale with the number ofatoms ranging from a few hundreds to a few thousands.In this paper, we introduce asemismooth Newton method that solves the dual problem of (NEDM). We prove that themethod is quadratically convergent.We then present an application of the Newton method to NEDM with $H$-weights.We demonstrate the superior performance of the Newton method over existing methodsincluding the latest quadratic semi-definite programming solver.This research also opens a new avenue towards efficient solution methods for the molecularembedding problem

Computing the nearest euclidean distance matrix with low embedding dimensions

Euclidean distance embedding appears in many high-profile applications including wireless sensor network localization, where not all pairwise distances among sensors are known or accurate. The classical Multi-Dimensional Scaling (cMDS) generally works well when the partial or contaminated Euclidean Distance Matrix (EDM) is close to the true EDM, but otherwise performs poorly. A natural step preceding cMDS would be to calculate the nearest EDM to the known matrix. A crucial condition on the desired nearest EDM is for it to have a low embedding dimension and this makes the problem nonconvex. There exists a large body of publications that deal with this problem. Some try to solve the problem directly and some are the type of convex relaxations of it. In this paper, we propose a numerical method that aims to solve this problem directly. Our method is strongly motivated by the majorized penalty method of Gao and Sun for low-rank positive semi-definite matrix optimization problems. The basic geometric object in our study is the set of EDMs having a low embedding dimension. We establish a zero duality gap result between the problem and its Lagrangian dual problem, which also motivates the majorization approach adopted. Numerical results show that the method works well for the Euclidean embedding of Network coordinate systems and for a class of problems in large scale sensor network localization and molecular conformation

Conditional quadratic semidefinite programming: examples and methods

The conditional quadratic semidefinite programming (cQSDP) refers to a class of matrix optimization problems whose matrix variables are required to be positive semidefinite on a subspace, and the objectives are quadratic. The chief purpose of this paper is to focus on two primal examples of cQSDP: the problem of matrix completion/approximation on a subspace and the Euclidean distance matrix problem. For the latter problem, we review some classical contributions and establish certain links among them. Moreover, we develop a semismooth Newton method for a special class of cQSDP and establish its quadratic convergence under the condition of constraint nondegeneracy. We also include an application in calibrating the correlation matrix in Libor market models. We hope this work will stimulate new research in cQSDP

A lagrangian dual approach to the single source localization problem

The single-source localization problem (SSLP), which is nonconvex by its nature, appears in several important multidisciplinary fields such as signal processing and the global positioning system. In this paper, we cast SSLP as a Euclidean distance embedding problem and study a Lagrangian dual approach. It is proved that the Lagrangian dual problem must have an optimal solution under the generalized Slater condition.We provide a sufficient condition for the zero-duality gap and establish the equivalence between the Lagrangian dual approach and the existing Generalized Trust-Region Subproblem (GTRS) approach studied by Beck et al. [“Exact and Approximate Solutions of Source Localization Problems,” IEEE Trans. Signal Process., vol. 56, pp. 1770–1778, 2008]. We also reveal new implications of the assumptions made by the GTRS approach. Moreover, the Lagrangian dual approach has a straightforward extension to the multiple-source localization problem. Numerical simulations demonstrate that the Lagrangian dual approach can produce localization of similar quality as the GTRS and can significantly outperform the well-known semidefinite programming solver SNLSDP for the multiple source localization problem on the tested cases

A convex matrix optimization for the additive constant problem in multidimensional scaling with application to locally linear embedding

The additive constant problem has a long history in multi-dimensional scaling and it has recently been used to resolve the issue of indefiniteness of the geodesic distance matrix in ISOMAP. But it would lead to a large positive constant being added to all eigenvalues of the centered geodesic distance matrix, often causing significant distortion of the original distances. In this paper, we reformulate the problem as a convex optimization of almost negative semidefinite matrix so as to achieve minimal variation of the original distances. We then develop a Newton-CG method and further prove its quadratic convergence. Finally, we include a novel application to the famous LLE (Locally Linear Embedding in nonlinear dimensionality reduction), addressing the issue when the input of LLE has missing values. We justify the use of the developed method to tackle this issue by establishing that the local Gram matrix used in LLE can be obtained through a local Euclidean distance matrix. The effectiveness of our method is demonstrated by numerical experiments

New sufficient conditions for global robust stability of delayed neural networks

In this paper, we continue to explore application of nonsmooth analysis to the study of global asymptotic robust stability (GARS) of delayed neural networks. In combination with Lyapunov theory, our approach gives several new types of sufficient conditions ensuring GARS. A significant common aspect of our results is their low computational complexity. It is demonstrated that the reported results can be verified either by conducting spectral decompositions of symmetric matrices associated with the uncertainty sets of network parameters, or by solving a semidefinite programming problem. Nontrivial examples are constructed to compare with some closely related existing result

Convex optimization learning of faithful Euclidean distance representations in nonlinear dimensionality reduction

Classical multidimensional scaling only works well when the noisy distances observed in a high dimensional space can be faithfully represented by Euclidean distances in a low dimensional space. Advanced models such as Maximum Variance Unfolding (MVU) and Minimum Volume Embedding (MVE) use Semi-Definite Programming (SDP) to reconstruct such faithful representations. While those SDP models are capable of producing high quality configuration numerically, they suffer two major drawbacks. One is that there exist no theoretically guaranteed bounds on the quality of the configuration. The other is that they are slow in computation when the data points are beyond moderate size. In this paper, we propose a convex optimization model of Euclidean distance matrices. We establish a non-asymptotic error bound for the random graph model with sub-Gaussian noise, and prove that our model produces a matrix estimator of high accuracy when the order of the uniform sample size is roughly the degree of freedom of a low-rank matrix up to a logarithmic factor. Our results partially explain why MVU and MVE often work well. Moreover, we develop a fast inexact accelerated proximal gradient method. Numerical experiments show that the model can produce configurations of high quality on large data points that the SDP approach would struggle to cope with